Touch sensitive computing surface for interacting with physical surface devices

ABSTRACT

Methods and systems for operating a touch sensitive display are disclosed. In one embodiment, a method for operating a touch sensitive display is provided that includes detecting a plurality of inputs via the touch sensitive display, interpreting, using at least one compute processor, each input in accordance with a location of the associated input on the touch sensitive display and a gesture of the associated input, generating a display based on the interpretation; updating a state of objects displayed on the touch sensitive display based on the interpretation, determining whether to generate an output to another device based on the interpretation, sending the output to the other device based on the determination, and displaying the generated display on the touch sensitive display.

CROSS-REFERENCE PARAGRAPH

This application claims priority to U.S. Patent Application No.61/872,934 filed on Sep. 3, 2013, the entire contents of which areherein incorporated by reference.

TECHNICAL FIELD

This disclosure relates generally to systems and methods for managingthe interaction between users, a touch sensitive computing surface, andphysical devices placed on the computing surface.

BACKGROUND

Touch screen devices that allow direct interaction with displays arebecoming increasingly prevalent in the marketplace. One type of touchscreen is called a “touch table” as the display is horizontal, allowingobjects to be placed on it, similar to any table or counter-top.Touch-sensitive devices may detect the presence and position offinger-based or object-based input, enabling the possibility of new waysto interact with electronic devices, or for electronic devices tointeract with each other.

The popularity of recent, small-scale touch screen devices, such asiPad® from APPLE®, means that touch screen devices may be found amongmany schools and family households, and the penetration of larger touchscreen systems in the marketplace is expected to increase.

SUMMARY

The present disclosure relates to systems and methods for operating atouch sensitive display and computing surface, including interactionswith users and physical surface devices placed on or near the computingsurface. The system may include a module to detect the state ofuser-operable controls located on the physical surface devices andwirelessly send corresponding data to modules that control the computingsurface. The system may further include a module to detect the locationof the physical surface devices and other touch or gesture inputs. Thisdata, in combination with the data corresponding to surface devicecontrols may be sent to a main surface device module. The main surfacedevice module, based on this data, may update the graphical display ofthe touch sensitive computing surface, update the menus and stateinformation stored by the computing surface, and the stored state of thephysical surface devices.

BRIEF DESCRIPTION OF FIGURES

FIGS. 1 and 2 illustrate a touch sensitive computing surface andphysical surface devices placed on top of the computing surface,according to some embodiments of the present disclosure;

FIG. 3 is a system diagram illustrating system components related tocommunications channels, according to some embodiments of the presentdisclosure;

FIG. 4 is a system diagram illustrating a physical surface device and anassociated touch sensitive computing surface, according to someembodiments of the present disclosure;

FIG. 5 is a flow chart illustrating the interaction and update sequencesthat may be executed during the operation of a touch sensitive computingsurface, according to some embodiments of the present disclosure;

FIG. 6 illustrates a configuration of wireless communications betweenphysical surface devices and a touch sensitive computing surface,according to some embodiments of the present disclosure;

FIG. 7 is a system diagram illustrating functionality of a touchsensitive computing surface, according to some embodiments of thepresent disclosure;

FIG. 8 illustrates a touch sensitive computing surface display,according to some embodiments of the present disclosure;

FIGS. 9 and 10 illustrate physical surface devices for interaction witha touch sensitive computing surface, according to some embodiments ofthe present disclosure;

FIG. 11 is a system diagram illustrating the functionality of a physicalsurface device, according to some embodiments of the present disclosure;

FIG. 12 is a flow chart illustrating a computerized method for receivinginput to a touch-sensitive computing device and making adjustments tothe display on the surface, according to some embodiments of the presentdisclosure;

FIG. 13 is a flow chart illustrating a computerized method forprocessing surface touches, according to some embodiments of the presentdisclosure; and

FIG. 14 illustrates is a flow chart illustrating a computerized methodfor processing and communicating control inputs of surface devicesplaced on a touch table, according to some embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The prevalence of large touch screen systems suggests that the touchtable environment may become a common working environment for a varietyof functions, providing responses to user inputs, reactions tomanipulation of objects on the touch surface, and multimedia informationand game components. In particular, the possibility of devicesspecifically engineered to interact and communicate with touch surfacesis anticipated.

In an entertainment setting, physical toys and battery-operatedelectronic toys remain standard playthings, with collectible figures andobjects remaining popular for imaginative toy play in the form ofinteraction between the figures and objects as well as the largerenvironment. In this context, kids are having increased exposure totouch screen devices making it desirable to provide new technologies,which may enhance the interactive experience with touch screen devicesand their existing toys.

Touch tables represent natural environments for intuitive and seamlessinterfacing of tangible toys and electronics with software ineducational and gaming contexts.

This disclosure describes user interface, software, and hardwareapproaches associated with the use and manipulation of objects on atouch computing surface or “touch table”. The use of location tracking,object designs, touch controls, and two-way wireless communications,either alone or in combinations, to create a seamless “virtual reality”in which tangible objects and instruments interact with a touch computeris described. Some embodiments have applications in educational gamesand laboratory experiments. For example, some embodiments may be used asin-classroom technology. Other embodiments additionally haveapplications in the non-educational entertainment sector.

As an example of an embodiment that may be used in an educationalsetting, one may imagine a group of students around a touch table.Through software, the table may guide the students to collide twophysical pucks together on the surface. If the students hit “start” onthe touch display and slide the pucks, the path of the puck may betracked and displayed. Furthermore, at the collision point of the pucks,vectors may be superimposed on the table display to illustrate velocityand conservation of momentum. The students may open up a graphicalcontrol panel for the puck and change its coefficient of friction andsee information about it. By touching the path, the students may displayplots of the displacement versus time, or overlay velocity versus timeand acceleration versus time. The students may also access the raw dataunderlying these displays. In this example, the specific hardware andsoftware combination allows the touch table to augment the reality of atangible teaching lab experiment and allows students to explore theequipment, graphical control panels, real-time data being generated, andmultimedia analysis.

As an example of the use of some embodiments in an entertainmentsetting, one may imagine interactive electronic action toys set onlarge-format, multi-player touch tables. Collectable figuresrepresenting characters, vehicles, game-pieces, or similar are trackedand identified on a large, iPad-like surface, becoming integrated intoan augmented video game. These figures may function as more than justinput devices. For example, two-way communications between the table andthe figures may allow video menu controls to change the function of thefigure electronics. For example, the communications may instruct thefigures to turn on and off lights, change sounds, or enable certainmodes. Plus, elements of the game software may be channeled through thefigures as output devices. For example, individual figures may beconfigured to call for the player's attention when something ishappening to them and indicate parameters from the game play. The use ofhollow or transparent elements in the game objects can also allowspecific viewing features of the underlying touch table graphics.

FIG. 1 is a multi-user touch table indicating the use of touch-drivensoftware and a microscope-shaped surface device to create a virtualexperiment, according to some embodiments. FIG. 1 includes users 102 and110, a physical surface device 104, a touch-sensitive computing surface106, and software displayed on the touch-sensitive computing surface108. In the figure, the software displayed on the touch-sensitivecomputing surface 108 is an interactive graphical display that may adaptto the position of both fingers and objects on the surface. Therefore,the touch table graphics image seen through the eyepiece of themicroscope may be expanded, blurred, or sharpened differently than thesurrounding graphics and in response to user inputs.

Users 102 interact with physical surface devices 104 placed on atouch-sensitive computing surface 106. The software displayed 108 on thetouch computer reacts to touch gestures from the users, placement of thesurface devices, and electronic communication with the surface devicesensors and input devices such as knobs and buttons. Additionally, thesurface device may have output modes (sounds, lights, displays)triggered by the software. Additional users 110 may interactsimultaneously with software touch interface components or additionalsurface devices. In the instance shown, a “virtual lab bench” is createdby using a surface object that looks like a microscope and allowsinspection of virtual objects with the software responding tomagnification and focus knobs. While the image viewed by the userthrough the microscope may be generated on the touch table under themicroscope surface object, the controls located on the microscope body(i.e., not physically attached to the touch table) may be used to modifythe image seen through the surface device. This is achieved by sendingwireless knob position data from the electronics of the microscopesurface object to the touch table which are then interpreted by thetouch table software. The resulting user experience is one ofcontrolling the viewing parameters of the microscope using controlslocated on the microscope, such as focus and zoom, though the changesare actually updated through the touch table software. Similarly, if themicroscope surface object is removed from the table, the display on thesurface may be modified by the touch table software, eliminating theimage that is intended to be seen through the microscope.

This embodiment is applicable to educational facilities and situationsin which real-world instrument skills may be taught efficiently usingvirtual materials. The use of embodiments for virtual microscopes,chemistry experiments using beakers and real reagents, projectile motionanalysis using real projectiles, and table-top robotics programmed withthe touch table are all examples of this kind of application.

FIG. 2 is a different application of the same technology for game play,according to some embodiments. FIG. 2 includes users 102 and 110,surface objects 204 on a touch table 106 with game elements displayed onits surface 208. Here, action figures are used on the table top andtheir positions and orientations may be tracked by the touch surface.The figures may be passive, simply tracked by properties or identifiersrecognized by the table, or active with batteries and internalcircuitry. In the latter case, the figures may include outputcomponents, such as movement, lights and sounds, and input components,such as touch sensors, buttons, movable limbs, or knobs.

In FIG. 2, the physical surface devices 204 take the form of toys,models, or action figures. The software displayed 208 on the touchcomputer reacts to touch gestures from the users, placement of thesurface devices, and electronic communication with the surface devicesensors and input devices such as knobs and buttons. Additionally, thesurface device may have output modes (movement, sounds, lights,displays) triggered by the software. Multiple users may interact withthe toys while the software handles the game environment and rules suchas attack modes, interaction modes, moves per turn, or attack range. Inaddition, touch table graphics may be visible in or through surfaceobjects through hollow, transparent, or other image conduits.

Therefore, from the figures it is shown that both input and output modesare distributed between the touch surface and the surface devices, whichmay interact using with and/or contact and wireless communications.

FIG. 3 is a system diagram illustrating system components related to thecommunications channels used to create a virtual reality experience,according to some embodiments. FIG. 3 includes representations of theuser interactions (302), the surface devices (304) including thephysical items (308) and their wireless communications capabilities(314), and also the touch table (306) including the actual computer(310) its software (312) and wireless capabilities (316). It isimportant to note that the user may naturally manipulate the surfaceobjects for some interactions and manipulate virtual touch controls onthe touch surface for others. In this figure elements managed by thetouch devices is shaded in gray and those managed by the surfacecomputer are shaded in blue.

FIG. 3 indicates that the users 302 interact with tangible surfacedevices (304) that are manipulated on a touch computer 306. The software312 displayed on the touch computer surface interacts via user inputgestures, the surface device location, and surface device input devices,such as knobs, buttons, and sensors. Surface device data may be sentfrom the surface device wireless communications channel 314 to the touchcomputer wireless system 316. Additionally wireless signals from thesoftware may trigger output modes of the surface devices such as sounds,lights, or sensing modes.

An element of the some embodiments is the plurality of interaction modesbetween the touch surface and electronic surface devices. Both positionand orientation may be tracked as well as more subtle input and outputmodes, the control information associated with some of the user inputsis passed by a wireless communications protocol, essentially invisibleto the user. This communication may be achieved with any of severalcommunications technologies and protocols including IR, RF, visiblelight, variable LED/LCD display markings, BlueTooth, ZigBee, WiFi orothers. Note again that the gray elements handled by the surface devicescommunicate with the blue elements handled by the touch computer, shadedin blue, via both touch sensing and wireless information paths.

FIG. 4 is a system diagram illustrating in more detail both thecommunications and essential elements of a surface device 402 and thetouch table 404, according to some embodiments. FIG. 4 includes arepresentation of the surface device 402 and its intrinsic capabilities406, 408, 410, 420, 424, and the touch table 404 and its intrinsiccapabilities 412,414,416,422,426. It is important to note that bothtouch table and surface device elements combine to create uniqueinteraction capabilities, and each may potentially be a simple orcomplex electronic device. Additionally, it is possible to use manysurface devices simultaneously on a single touch table, though this isnot indicated in the figure. FIG. 4 includes the surface device 402including inputs 406 and outputs 408 of the physical item 410 which alsoincludes internal firmware 420 and wireless communications capabilities424 and also the touch table 404 including display surface 412 andperipherals 414 of the main computer 416 which also has internalsoftware 422 and wireless communications 426.

In FIG. 4 the surface devices 402 interact with the touch computer 404to create an interactive, tangible experience for the user. Thelocation, state, inputs 406, and outputs 408 on the surface device body410 are perceived by the user while observing the touch computer display412 and other touch computer input and output peripherals 414 such asspeakers and video game controllers. These are controlled by the mainsoftware processor 416 which runs application software 422 withgraphical elements such as a game or teaching application. The state ofthe surface device is interactive with the touch computer and itssoftware through both wireless communication 424 and 426 and directinterpretation of the surface device location and size on the surface.

In this figure, a limited set of communication modes, namely a touchscreen and wireless communications, enables a very broad array of inputand output modes from multiple sources and encompassing multiple sensesand multiple surface devices may be used identically at the same time,enabling extensive multiplexing of input and output modes. An example ofthis is a chess game in which each game piece may independently createdisplay and sound interaction with the players. Moreover, the appearanceand role of the game pieces may be established through the touch tablegraphics, either around the game piece, or visible through a transparentor hollow game piece.

FIG. 5 is a flow chart illustrating a real-time interaction and updatesequence that may be executed during the operation of a surface device,according to some embodiments. User inputs and reactions 532 areimportant, and drive detection 502 and update routines 524, 526, and 528that are distributed between the surface devices and theirelectronics/firmware and the touch surface and its electronics/firmware.Many surface devices may be used and the essential update loop isunchanged.

As shown in FIG. 5, the main program 530 provides graphics 528interactive menus 526 and the current surface device control 524 inreal-time. Observing the tangible elements and system outputs, the usermay interact 532 with the touch computer/surface device system in avariety of ways 512-522. Upon the user's observing the software outputsand adjusting controls or locations, the surface device senses changesto controls 502 and the touch computer senses changes to locations 506and touch inputs 508, and these are sent to the software either throughwireless link 504 or direct hardware communications 510. The participantinteraction is then interpreted by the main program 530, restarting thereal-time interaction loop.

As an example of the flow of use described by FIG. 5, consider a userthat approaches an operating touch table with two activeaction-figure-type surface devices on its top. The user observes thepositions and poses of the action figures 518 and also the underlyinggraphics 522 on the touch table around and under the figure thatillustrates possible movements or explorations available to the user.Perhaps a sound 520 invites the user to move one of the surface devices.The user moves the figure along the surface 514 and also adjusts thefigure's pose 512 and moves graphical elements displayed on the touchtable with his fingers 516. Invisible to the user, the touch tabledetects the changes in the figure 506 and also the changes to the actionfigure's pose 502 which in this example may be raising the figure'ssword, which is sent by wireless communications 504 to the touch table.All of the data associated with the user interactions are thereforetransmitted 510 to the touch table software 530, which is responsiblefor updating the table graphics 528 such that a new portion of the gamemap and a graphical adversary is revealed on the display, and updatesthe game mechanics 526 such that since the figure's sword is now raised,a particular touch gesture may signify an attack and trigger an outputfrom the surface devices 524 such that a speaker in the action figuresays “Chop!” and a speaker in the other action figure, which was nevermoved says “Watch out!”. Completing the loop to the participant 532,these changes in graphics and output are observed 518, 520, 522 and theuser may move or adjust the action-figure surface objects again 512,514, expecting a continuation of the game play through new graphics andoutputs and new manipulation possibilities for his hands on the touchtable 516.

FIG. 4 demonstrates the update loop from the center to the top, to thebottom and to the center, to repeat. With modern computing devices, thisloop may be completed at a rate faster than the user-interaction speed(perhaps 40 Hz). Similarly, with even simple wireless communicationstechnologies (IR, BlueTooth, ZigBee, etc.) enough data may be movedthrough the wireless link to allow closely-coupled, real-time input andoutput from the surface devices and the touch display. As in otherfigures, elements that are principally handled by the surface devicesare shaded in gray and elements handled by the surface computer areshaded in blue.

In some embodiments the wireless communications may operate withoutdirect intervention of the user. This creates a combined reality effect,such as a game board recognizing game pieces which, in turn light upwhen placed on the table.

FIG. 6 indicates one layout of the wireless communications, according tosome embodiments. Users 602 and 608 interact with surface devices 604through manipulation of location, controls, and responses. Touch inputs606 also may be used to interact with graphical software elements. Thetouch computer 610 senses changes to surface objects that are broadcastthrough a wireless technology such as IR or RF 614 and received by thecomputer 612.

A variety of wireless communications technologies may be applied to thepresent disclosure including IR, optical, or RF (including BlueTooth,WiFi, or ZigBee). The present disclosure has the ability to createinnovative user interface concepts through the combination of wirelessconnectivity and touch devices, especially with respect to the additionof wireless-enabled surface devices on top of the touch screen. It mayalso be noted that surface object augmented reality effects under thisdisclosure may be created without wireless communications throughmanipulation of touch and position detected more directly by the touchtable.

FIG. 7 is a system diagram illustrating functionality of the touchcomputer, according to some embodiments. The touch computer used in thesystem may have a variety of components and functions. The touch/displaysurface 700 is the center of user interaction. This is attached via achassis 702 to the main processor 706 which runs the interactiveapplication. The software may use additional input sources 704, 710, and714, such as game controllers, mouse, track ball, or position sensors,and may also provide additional outputs 708, 712, and 716, such asspeakers, lights, vibration, projectors, or fog, in addition to thegraphical output of the touch surface around and through surfaceobjects. As an example, a wireless controller like a track ball may beused both as a control device using position on the touch table, similarto a mouse, and also provide interaction using the ball itself andbuttons on the controller. Output modes may include vibration of thetrack ball controller as well as changing projected or illuminateddisplays, plus graphics on the table top around the track ball, perhapsindicating options, or even through a transparent track ball, changingcolor and labeling of input modes. A wireless communications system 718can provide two-way connectivity to the surface devices manipulated onthe touch surface to both sense state changes in the surface devices andalso trigger state changes from the software.

The importance of the functionality described in FIG. 7 is to establishkey elements that may be combined in some embodiments to create avirtual reality experience using the touch computer as an interactivetable. This level of interactivity arises from the interplay of thegraphics displayed by the touch computer and surface devices placeddirectly on these images and affecting them in real time.

FIG. 8 is an example of the display that might be visible on the surfacecomputer, according to some embodiments. FIG. 8 includes the displayscreen 802, under-object graphics 804, other control graphics 806,including material selector buttons 808. Graphical software displayed onthe touch computer may include graphical elements 804 that track andmove with the surface devices and change according to surface devicecontrols (such as zoom and focus). Additionally, touch-activated menus806, and multi-user controls and views may be supported for interactivegroup or team use.

Note that the graphics indicated in FIG. 8, may not be stationary andunchanging, but rather part of an interactive computer application suchas a game or analytical program that changes in response to useractions. In particular, the actions of the user to move and adjust thesurface devices placed on the touch computer surface.

FIG. 9 is a microscope-shaped surface device, according to someembodiments. FIG. 9 includes microscope control knobs 902, themicroscope body that may be looked through 904, an indication ofinternal electronics 906, and the object's base 908 that rests on thetouch table. The physical surface devices 904 may take the form of toolsor devices with control knobs 902 that affect the software in a mannerthat simulates the control functionality of physical systems, such as amicroscope. Internal electronics 906 sense and translate controladjustments to wireless data that is received and interpreted by thetouch computer which reflects the changes in the software graphics. Thesoftware may also send wireless messages to the surface device to changestate and outputs, such as lights and sounds. Additionally, the size,orientation, and location of the surface device base 908 can be trackedby the touch computer, allowing tangible control over software elements.

Using a surface device like this one, virtual laboratory experiments arepossible that sharpen a student's skills using the controls of reallaboratory equipment, but taking advantage of a large digital imagelibrary, thereby speeding up sample preparation and making educationless expensive. As the focus and positioning knobs on the surface deviceare changed, the software reacts by modifying the image seen through theeyepiece, which is actually displayed on the touch computer screen. Thisis one example out of many educational possibilities including usingbeakers, heaters, mechanical systems, and robots as surface devices.More basic educational applications include letter or number blocks,especially transparent blocks that provide physical manipulation, butthe appearance of which is goverened by the underlying touch tablegraphics that track and move with the blocks. Early educationapplications include math, spelling, sorting, pattern matching, andsentence formation.

In addition to educational applications for surface devices, there are avariety of possibilities for entertainment. Current toy products thatcombine tangible action figures with computer software have very limitedinteraction possibilities with regards to manual manipulation of theaction figures. Generally the figure is placed or attached to thecomputer system and the gameplay is essentially identical to asoftware-only game. The surface device design can allow additionalmanipulation of the figure to affect game play and be closely integratedwith the software. Additionally, traditional board game concepts can beaugmented with game pieces that change label or color throughtransparency, or with additional menu elements to allow usermanipulation of game piece identity or capabilities.

FIG. 10 shows examples of action figure surface devices, according tosome embodiments. The physical surface devices 1004 may take the form oftoys, models, or figures with control elements such as a pivoting arm1006 that affects the software in gameplay. Internal electronics 1002sense and translate control adjustments to wireless data that isreceived and interpreted by the touch computer which reflects thechanges in the software graphics around and beneath surface objects. Thesoftware may also send wireless messages to the surface device to changestate and outputs, such as lights and sounds evocative of gameplay.Additionally, the size, orientation, and location of the surface devicebase 1008 are tracked by the touch computer, allowing tangible controlover software elements.

The wireless connectivity between the surface devices and the touchcomputer is not necessary, but enables additional interactivepossibilities. The mechanical manipulation of the surface devices on thetouch computer and responses of the electronics in the surface devicesplay an important role in the interactivity of the device. For example,the surface device may simply react to the display color underneath itor make sounds and display outputs randomly, to which the user responds,thereby affecting the software.

FIG. 11 is a system diagram illustrating the intrinsic functionality ofone surface device, according to some embodiments. The surface devicesused in the system may have a variety of components and functions. Thedevice chassis 1106 enables physical manipulation that is tracked by thetouch computer when place on its base 1104. The chassis may also includemovable parts 1104 which trigger software responses. Inside the surfacedevice, a processor 1108 tracks the state of the device and controls itsfunctions. The surface device may use additional input sources 1110,1114, and 1118, such as touch screens, light sensors, position sensors,cameras, or microphones, and may also provide additional outputs 1112,1116, and 1120, such as sound, vibration, lights, integrated displays,and movement, beyond the tracked, physical presence of the surfacedevice on the touch surface. A wireless communications system 1122provides two-way connectivity to the touch computer.

The manipulation of electronic surface devices place directly on top ofa horizontal touch computer is central to some embodiments. No singleattribute indicated above is critical to achieving the uniqueinteraction functions described as a “augmented reality” experience. Insome embodiments, the present disclosure has the ability for the touchsurface to react in an intuitive way to manipulations of the objects seton it.

FIG. 12 is a flow chart illustrating a computerized method for receivinginput to a touch-sensitive computing device and making adjustments tothe display on the surface, according to some embodiments. FIG. 12includes software processes for receiving the surface input data fromboth finger touches and objects on the touch table, interpreting thisdata (1204) in the context of the software program or game currentlyactive, generating an updated surface display (1206) in response to thenew data, and generating updated states of the objects on the table(1208) such as internal firmware state, lights, or sounds.

At step 1202, data representing touch gestures from users and describingsurface devices placed on the touch-sensitive computing device isreceived. While different touch table technologies receive position andtouch information in different ways (capacitive, optical, etc.) thisprocessing step represents converting all raw touch table data toposition, size, and orientation data suitable for use by the mainprogram. As an example, the user may move action-figure-like surfaceobjects, change their poses, and touch menus and graphical elements withtheir fingers. All of this manipulation can be converted intosoftware-actionable data in this step such that the program may processthe next step in the interaction, changing the image, such as a map orgame board displayed, opening menus and pull-downs, and updatingspecific statistics of the game character represented by the figure,such that a particular weapon or action is indicated in the softwarethat was suggested by a modified pose, for instance.

At step 1204 the data is interpreted with regards to controlling ormodifying the application software. As described above, once the rawtouch table and surface object data is received and converted intosoftware-actionable information, the main program may interpret the datato discern user input intentions and suitable real-time responses. Forexample, if a user finger touch on a menu graphic is received in step1202, in step 1204 processes in the main program may interpret thisinformation in the context of the program to trigger intended results,such as opening up a graphical menu. This interpretation can be specificto the particular program running, and it is not necessary that everyuse of a given hardware implementation have the same response to touchand surface object inputs. For example, a game about boats mightinterpret an illegal move if the surface object is moved out of an areadisplaying water, where a game about kittens may do the opposite,interpreting that the surface object should not be in the virtual water.Illegal moves can be indicated to the user by changing the color ofgraphics around or under the game piece, or restricting the movement ofa graphic that tracks the physical piece, requiring the player tobacktrack to replace the piece on its representative graphic.Transparent pieces allow color and labeling changes to be displayed onthe table and observed in or through the game piece.

At step 1206 the software display is updated, modifying the touchsurface appearance in conjunction with the original user input. Thismeans that the software-actionable data received and interpreted inprior steps is actually indicated to the user on the touch table displayand other outputs in step 1206. Continuing previous examples, in thisstep a graphical menu may be displayed as “open” in response to theappropriate touch event, and an “x” or buzzer sound might indicateillegal moves in the example of a game in which water obstacles may beobserved. It is important to note that the touch table may have multipleoutput modes that are triggered in this step including the main display(creating a red “x” under or around the surface device or touchlocation), speakers (creating a buzzer sound), additional displays (suchas moving an image to a projected display on the wall in response to agesture), and external lights (such as dimming the room lights inresponse to a gesture or surface object rotation).

At step 1208 the state of the surface devices is updated providing theuser with additional stimuli arising from the initial input. This stepmay involve the use of active surface devices that provide anotherdimension of user interaction. In the case that the surface objects arenot merely inanimate objects tracked and interpreted by the touch table,electronic surface devices may themselves have internal software statesand output modes that may be updated. Continuing the above examples, anelectronic cat surface device may itself emit a “distressed meow” soundif placed over a graphic of water. This allows greater localization andvirtualization of the game play. Output modes may include surfacedevices that make sounds or vibrate in response to specific movementsrelative to touch table graphics. Additionally, the surface device maybe transparent or may itself have a display that reacts to the touchtable program, such as a transparent block, smart phone, or tabletdisplaying a compass needle that reacts to virtual magnets distributedon the touch table screen. The magnet reading may spin if the object isremoved from the surface, and change when positioned on the surface toindicate the magnetic field that may result from the displayedconfiguration.

FIG. 13 is a flow chart illustrating a computerized method forprocessing surface touches, according to some embodiments. This figureprovides additional detail not shown in FIG. 12, specifically relatingto the operations performed by the main processor associated with thetouch table. FIG. 13 includes processes to detect surface touches andobject locations 1302, conveying this data to the main application 1304,updating the touch table surface display 1306, generating updates to alllocal output devices 1308, and finally sending updates to all surfaceobjects and non-local output devices 1310. The blue boxes indicate thatall of these processes are performed by, or scheduled by, the mainprocessor, typically associated with the touch table.

At step 1302, the touch table detects user touches and gestures as wellas the location of surface devices placed on the touch surface. This isdescribed in more detail in association with FIG. 12, since the touchtable is generally the main hardware system detecting touches and objectlocations on its surface. It is not necessary that the main processor byincorporated into, or even physically attached to the touch table, assome touch surfaces are capable of sending raw or processed data to anancillary processor, possibly controlling many such touch surfaces. Theprimary intention of the figure is to indicate that at some point thetouch information needs to be detected and packaged for transmission tothe main program to be interpreted and drive the user interaction.

At step 1304, both user touches and surface device location data isconveyed to the main program along with any data received from thesurface devices themselves. As noted above, the transmission of the datamay be within a single device or networked among many devices eitherwith wires or wirelessly. There may be many touch surfaces deliveringinformation to one processor, or there may be many computers allreceiving information from a single touch surface, or any othercombination. The intention of this step is to establish that there needsto be a process by which the main application responsible for the userinteraction receives the touch information in a format that isprogram-actionable and may trigger appropriate interface responses, suchas updating graphics, opening menus, changing game boards, etc.

At step 1306, the surface display is updated according to the software.Since in some embodiments, the display represents a changeable gameboard or virtual desktop, changing the touch table display is importantto the user interface, and generally is done in direct response to userinputs, similar to moving windows and icons graphics in response tomouse controls in a traditional computer interface design. Unlike atraditional, one-mouse interface, in the described system, multipletouches, positions, and inputs from many hands, objects, and users maybe processed at the same time, making the system very collaborative insome embodiments. Step 1306 does not presuppose what kinds of displaysor graphics are updated or how, but rather this may be specific to theinterface and software application being executed. Step 1306 indicatesthat in the described method there may generally be a change in thegraphics displayed on the touch table to indicate the user interfaceintention and provide feedback to the user.

At step 1308, any other outputs, such as sounds, vibrations, oradditional lights are triggered. Some additional output modes may belocal to the touch table system or directly controlled by this system,regardless of the particular architecture of touch-sensitive devices andprocessors and communications. These are distinct from outputscontrolled by other, remote processors, such as the surface objects, butmay still be triggered by, or in communication with the mainapplication. Examples of the processes represented in step 1308 includesounds from speakers in or around the touch table, other displaysdirectly controlled by the main process, such as a projected screennearby, or even the release of fog or scents triggered by the userinterface.

At step 1310, updated state data for the surface objects and devices seton the touch surface are conveyed. Though an active surface device mayhave its own processor that interprets inputs and provides independentoutputs, generally there may be a level of communication between thesesemi-independent systems and the main application to create a unifiedinterface experience. As an example, the data sent from the main processto a surface device may only be state information that does not directlytrigger any perceptible change. One case is an action figure with acertain amount of strength or life remaining in the game. This mayupdate the state such that if the figure is removed from the table, itmay report its “life value” at any other time through the push of abutton. Alternately, this life value might trigger an output such as asound when some other game process sends data, such as when a particular“report health” button is pressed on the touch table screen, all of thesurface figures on the touch table may all report their health, perhapssaying “I feel great!” or “I need food badly”.

The entire process represented in FIG. 13 may be viewed as a loop thatexecutes in real time. The application may be turn or instance-driven,or may be changing in real-time independent of user interactions.

FIG. 14 is a flow chart illustrating a computerized method forprocessing and communicating control inputs of surface devices placed ona touch table, according to some embodiments. FIG. 14 includes stepsthat process control inputs local to the surface device 1402, sendrelevant data regarding the surface device state to the main applicationprocess 1404, receive new information regarding other user inputs andthe main application 1406, and perform any output functions required aspart of the user interface 1408. The entire process represented in FIG.14 may be viewed as a loop that executes in real time. Updates may beturn or instance-driven, or may be changing in real-time independent ofuser interactions.

At step 1402, adjustments to direct controls on the surface devices areinterpreted. These controls may be knobs and buttons on an otherwiseunrecognizable object, for example if localized control was needed thatchanged function depending where it is on the table. An instance of thisis a knob that changes the color of blocks beneath it and may be movedfrom block to block, and the knob repeatedly adjusted. In this case, theknob may always changes color, but the block it is addressing may bespecific to the controller's location on the table and the specificgraphics currently being displayed. If the user changes the arrangementof blocks using finger drag gestures, the position required to changethe block's color may likewise be adjusted. The local controls onsurface objects may also be representative of recognizable instruments,characters, or game-play actions. For example, if the surface object isin the form of an action-figure, changing moveable elements like armpositions or a drawn weapon may be sensed by local electronics in thesurface object and become part of the user interface. Similarly, asurface object that looks like a microscope may have zoom and focusknobs similar to a real microscope that create interface effects thatmimic these optical effects.

At step 1404, this data is conveyed to the main program to join thetouch, location, and orientation data to determine the softwareresponse. As part of this step, the inputs applied by the user to thesurface objects may be communicated to the application that updates thegraphics and outputs according to the intended interface experience. Asan example, a surface object that is in the shape of an action figuremay have moving parts that affect the game play represented on thescreen. An instance of this might be a toy dragon, that when you pressits head it temporarily lights up red using LED illuminators embedded inthe toy. The status of the dragon as “in the red state” may be sent tothe main application, if a corresponding fire effect is to be displayedon the touch table display around or under the figure.

At step 1406, updated state information is received by the surfacedevices from the main program. Continuing the dragon toy example fromthe step above, if the “red state” control is activated and thisinformation is interpreted by the main program to fill an area of thedisplay with a fire effect, there may be other surface objects incontact with the resulting virtual fire that may also have their statechanged as a result. Perhaps another dragon figure some distance awayalso turns red when in contact with a fire graphic. This state change tored of the second dragon may be triggered by receiving information fromthe main program. Therefore, the internal state variable that definesthe dragon as “red” or “not red” can be modified by receiving wirelessinformation from the touch table.

At step 1408, these trigger the intended output modes of the surfacedevices such as lights, displays, and sounds. This step represents theactual change of tangible interface elements local to a surface objectbased on the software application. In the prior “red dragon” example,this may include actually changing the state of the second dragon to“red” in response to being touched by the flame graphic. Other outputmodes may include indicator lights on the surface object that show theuser the current internal state of the device, such as “the knob nowchanges color of blocks below the surface object” versus “the knob nowchanges size of blocks below the surface object”. As describedpreviously, surface object output modes may include sounds, lights,vibrations, movements, or other tangible changes local to the surfaceobjects.

The subject matter described herein may be implemented in digitalelectronic circuitry, or in computer software, firmware, or hardware,including the structural means disclosed in this specification andstructural equivalents thereof, or in combinations of them. The subjectmatter described herein may be implemented as one or more computerprogram products, such as one or more computer programs tangiblyembodied in an information carrier (e.g., in a machine readable storagedevice), or embodied in a propagated signal, for execution by, or tocontrol the operation of, data processing apparatus (e.g., aprogrammable processor, a computer, or multiple computers). A computerprogram (also known as a program, software, software application, orcode) may be written in any form of programming language, includingcompiled or interpreted languages, and it may be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program does not necessarily correspond to a file. A programmay be stored in a portion of a file that holds other programs or data,in a single file dedicated to the program in question, or in multiplecoordinated files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program may be deployed to beexecuted on one computer or on multiple computers at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification, includingthe method steps of the subject matter described herein, may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions of the subject matter describedherein by operating on input data and generating output. The processesand logic flows may also be performed by, and apparatus of the subjectmatter described herein may be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processor of any kind of digital computer. Generally, aprocessor may receive instructions and data from a read only memory or arandom access memory or both. The essential elements of a computer are aprocessor for executing instructions and one or more memory devices forstoring instructions and data. Generally, a computer may also include,or be operatively coupled to receive data from or transfer data to, orboth, one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. Information carriers suitablefor embodying computer program instructions and data include all formsof nonvolatile memory, including by way of example semiconductor memorydevices, (e.g., EPROM, EEPROM, and flash memory devices); magneticdisks, (e.g., internal hard disks or removable disks); magneto opticaldisks; and optical disks (e.g., CD and DVD disks). The processor and thememory may be supplemented by, or incorporated in, special purpose logiccircuitry.

It is to be understood that the disclosed subject matter is not limitedin its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The disclosed subject matter is capable ofother embodiments and of being practiced and carried out in variousways. Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting.

As such, those skilled in the art will appreciate that the conception,upon which this disclosure is based, may readily be utilized as a basisfor the designing of other structures, methods, and systems for carryingout the several purposes of the disclosed subject matter. It isimportant, therefore, that the claims be regarded as including suchequivalent constructions insofar as they do not depart from the spiritand scope of the disclosed subject matter.

Although the disclosed subject matter has been described and illustratedin the foregoing exemplary embodiments, it is understood that thepresent disclosure has been made only by way of example, and thatnumerous changes in the details of implementation of the disclosedsubject matter may be made without departing from the spirit and scopeof the disclosed subject matter.

What is claimed:
 1. A method for operating a touch sensitive integrateddisplay system comprising: detecting a plurality of multiuser inputs viathe touch integrated sensitive display system; interpreting, using atleast one compute processor, each input in accordance with a location ofthe associated input on the touch integrated sensitive display systemand a gesture of the associated input; generating a display of agraphical image based on the interpretation; updating a state of objectsdisplayed on the touch sensitive integrated display system based on theinterpretation; determining whether to generate an output to another adevice comprising an eyepiece that interacts with and is separate fromthe touch sensitive integrated display system based on theinterpretation; sending the output to the other separate device based onthe determination; and displaying the generated display of the graphicalimage on the touch sensitive integrated display system which is viewablethrough the eyepiece.
 2. The method of claim 1, further comprising:tracking a position of the separate device relative to the touchsensitive display; and updating the state of the objects displayed onthe touch sensitive display based on the tracked position.
 3. The methodof claim 2, wherein the determination of whether to generate the outputto the separate device is based on the position of the separate devicebeing tracked to a predetermined position relative to the touchsensitive display.
 4. The method of claim 1, further comprising:tracking an orientation of the separate device relative to the touchsensitive display; and updating the state of the objects displayed onthe touch sensitive display based on the tracked orientation.
 5. Themethod of claim 1, further comprising: updating the state of the objectsdisplayed on the touch sensitive display in response to movement of theseparate device while in contact with the touch sensitive display. 6.The method of claim 1, further comprising: generating an output from oneof a speaker and a light based on the interpreted inputs.
 7. The methodof claim 1, wherein the interpreting comprises determining a position,size, and orientation of each input relative to the touch sensitivedisplay.
 8. The method of claim 1, wherein the separate device and thetouch sensitive table communicate via a wireless communication protocol.9. The method of claim 8, wherein the wireless communication protocol isone of infrared, radio frequency, visible light, variable LED/LCDdisplay markings, Bluetooth, ZigBee, and Wi-Fi.
 10. The method of claim1, wherein the separate device is at least one of a toy, model, andaction figure.